Offensive Security Researcher & Quantum Cryptography Analyst

How to Set Up Kubernetes Monitoring with Kibana

Abdul Wahab Junaid
How to Set Up Kubernetes Monitoring with Kibana

When people say “Kubernetes monitoring,” they usually mean metrics — Prometheus, Grafana, dashboards full of CPU graphs. But logs tell a different, often more useful story during an actual incident, and that’s where Kibana comes in. I lean on the Elastic Stack (Elasticsearch, Kibana, and either Filebeat or Fluent Bit) whenever I need to search across thousands of pod logs during a postmortem, and setting it up correctly on Kubernetes has a few gotchas worth knowing upfront.

Kibana’s Role in the Stack

Kibana itself is just a visualization and query UI — it doesn’t collect or store logs. The full pipeline looks like this:

Pods (stdout/stderr logs)
        │
        ▼
Node-level log collector (Filebeat / Fluent Bit DaemonSet)
        │
        ▼
Elasticsearch (storage + indexing)
        │
        ▼
Kibana (search, dashboards, alerting)

Kubernetes writes container stdout/stderr to files on the node (/var/log/containers/*.log via the container runtime), and a DaemonSet-based log shipper tails those files, enriches them with Kubernetes metadata (pod name, namespace, labels), and forwards them to Elasticsearch.

Deploying Elasticsearch and Kibana

I use the Elastic Cloud on Kubernetes (ECK) operator for this — it manages the Elasticsearch and Kibana lifecycle far better than hand-rolled StatefulSets.

kubectl create -f https://download.elastic.co/downloads/eck/2.13.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.13.0/operator.yaml

Verify the operator is running:

kubectl get pods -n elastic-system

Now define an Elasticsearch cluster:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logging-es
  namespace: logging
spec:
  version: 8.13.0
  nodeSets:
    - name: default
      count: 3
      config:
        node.store.allow_mmap: false
      volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 50Gi
            storageClassName: gp3

And Kibana:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: logging-kibana
  namespace: logging
spec:
  version: 8.13.0
  count: 1
  elasticsearchRef:
    name: logging-es
  http:
    tls:
      selfSignedCertificate:
        disabled: true

kubectl create namespace logging
kubectl apply -f elasticsearch.yaml
kubectl apply -f kibana.yaml
kubectl get elasticsearch,kibana -n logging

Retrieve the auto-generated elastic user password:

kubectl get secret logging-es-es-elastic-user -n logging -o jsonpath='{.data.elastic}' | base64 -d

Access Kibana:

kubectl port-forward -n logging svc/logging-kibana-kb-http 5601:5601

Shipping Logs with Filebeat

Deploy Filebeat as a DaemonSet so every node has an agent tailing container logs:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: logging-filebeat
  namespace: logging
spec:
  type: filebeat
  version: 8.13.0
  elasticsearchRef:
    name: logging-es
  kibanaRef:
    name: logging-kibana
  config:
    filebeat.autodiscover.providers:
      - type: kubernetes
        node: ${NODE_NAME}
        hints.enabled: true
        hints.default_config.type: container
        hints.default_config.paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log
    processors:
      - add_cloud_metadata: {}
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        containers:
          - name: filebeat
            securityContext:
              runAsUser: 0
            volumeMounts:
              - name: varlogcontainers
                mountPath: /var/log/containers
              - name: varlogpods
                mountPath: /var/log/pods
              - name: varlibdockercontainers
                mountPath: /var/lib/docker/containers
        volumes:
          - name: varlogcontainers
            hostPath:
              path: /var/log/containers
          - name: varlogpods
            hostPath:
              path: /var/log/pods
          - name: varlibdockercontainers
            hostPath:
              path: /var/lib/docker/containers

RBAC for Filebeat’s Kubernetes metadata enrichment:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: filebeat
rules:
  - apiGroups: [""]
    resources: ["namespaces", "pods", "nodes"]
    verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
  - kind: ServiceAccount
    name: filebeat
    namespace: logging
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io

kubectl apply -f filebeat.yaml
kubectl get pods -n logging -l beat.k8s.elastic.co/name=logging-filebeat

Creating Index Patterns and Dashboards in Kibana

Once logs are flowing, open Kibana and go to Stack Management → Index Patterns, create a pattern matching filebeat-*, and select @timestamp as the time field.

From the Discover tab, I typically build saved searches filtered by kubernetes.namespace and kubernetes.pod.name so on-call engineers can jump straight to a specific workload’s logs during an incident.

For dashboards, useful visualizations include:

Kubernetes-Native Alerting

Kibana’s alerting framework (under Stack Management → Rules) can watch a query — for example, error log count exceeding a threshold in a 5-minute window — and fire a webhook, email, or Slack notification.

{
  "params": {
    "index": ["filebeat-*"],
    "esQuery": "{\"query\":{\"bool\":{\"must\":[{\"match\":{\"log.level\":\"error\"}}]}}}",
    "threshold": [50],
    "thresholdComparator": ">",
    "timeWindowSize": 5,
    "timeWindowUnit": "m"
  }
}

Securing Access to Kibana and Elasticsearch

Logs frequently contain sensitive data — auth tokens accidentally logged, PII in request bodies, internal hostnames — so locking down access matters as much as getting the pipeline working. ECK enables basic auth by default, but for real production use I layer in role-based access control at the Elasticsearch level:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: logging-es
  namespace: logging
spec:
  version: 8.13.0
  auth:
    roles:
      - secretName: es-roles
  nodeSets:
    - name: default
      count: 3

apiVersion: v1
kind: Secret
metadata:
  name: es-roles
  namespace: logging
stringData:
  roles.yml: |
    read_only_logs:
      indices:
        - names: ["filebeat-*"]
          privileges: ["read"]

For Kibana itself, put it behind an Ingress with TLS and integrate with your organization’s SSO provider (SAML or OIDC) rather than relying on shared basic-auth credentials passed around a team:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kibana-ingress
  namespace: logging
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/auth-url: "https://auth.internal.example.com/verify"
spec:
  tls:
    - hosts:
        - kibana.internal.example.com
      secretName: kibana-tls
  rules:
    - host: kibana.internal.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: logging-kibana-kb-http
                port:
                  number: 5601

Index Lifecycle Management in Detail

I mentioned ILM briefly above, but it deserves a fuller example since misconfigured retention is the single most common cause of Elasticsearch clusters running out of disk mid-incident — exactly when you need logs most.

PUT _ilm/policy/filebeat-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "30gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "3d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Apply this via Kibana’s Dev Tools console or curl against the Elasticsearch service directly, then bind it to the filebeat-* index template so every new daily index picks it up automatically.

Troubleshooting the Log Pipeline

Logs not appearing in Kibana at all. Work backward through the pipeline:

kubectl logs -n logging -l beat.k8s.elastic.co/name=logging-filebeat --tail=50
kubectl exec -n logging <filebeat-pod> -- filebeat test output

Logs appear but Kubernetes metadata (pod name, namespace) is missing. This almost always means the add_kubernetes_metadata processor isn’t matching correctly — verify the logs_path matcher aligns with your container runtime’s actual log path, which can differ between containerd and Docker-based nodes.

Elasticsearch cluster status is yellow or red. Check shard allocation:

kubectl exec -n logging logging-es-default-0 -- curl -s -u elastic:$PASSWORD localhost:9200/_cluster/health?pretty

Yellow typically means replica shards are unassigned (common with a single-node cluster where replicas can’t be placed); red means primary shards are missing, which is a genuine data-loss risk requiring immediate investigation.

Production Considerations

Handling Multiline Logs (Stack Traces)

By default, Filebeat treats each line as a separate log event, which shreds multi-line stack traces into dozens of disconnected entries in Kibana — one of the most common early frustrations with this stack. Fix it with a multiline processor keyed on a pattern that identifies the start of a new log entry, typically a timestamp:

filebeat.autodiscover.providers:
  - type: kubernetes
    node: ${NODE_NAME}
    hints.enabled: true
    hints.default_config.type: container
    hints.default_config.paths:
      - /var/log/containers/*${data.kubernetes.container.id}.log
    hints.default_config.multiline.pattern: '^\d{4}-\d{2}-\d{2}'
    hints.default_config.multiline.negate: true
    hints.default_config.multiline.match: after

With negate: true and match: after, any line that does NOT match the timestamp pattern gets appended to the previous log entry rather than treated as its own event — exactly what you want for a Java or Python stack trace spanning many lines.

Structured Logging for Better Kibana Queries

Raw text logs work fine for Discover tab searching, but structured JSON logs unlock far more powerful filtering and visualization. If your application can emit JSON directly to stdout, Filebeat can parse it automatically:

console.log(JSON.stringify({
  level: "error",
  message: "Payment processing failed",
  order_id: "ord_12345",
  timestamp: new Date().toISOString()
}));

processors:
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true

This turns free-text log lines into queryable fields (order_id: "ord_12345"), which is dramatically more useful in Kibana than string-matching against raw text, especially for building dashboards that need to aggregate by a specific field rather than just count occurrences of a substring.

Common Mistakes

Summary

Kibana gives you the searchable, human-readable side of Kubernetes observability that raw metrics can’t provide. The real setup work is in getting Filebeat’s Kubernetes metadata enrichment right and managing Elasticsearch’s lifecycle so it doesn’t fall over under log volume. Once that pipeline is solid, Kibana becomes the tool I reach for first during any incident that needs “what actually happened, in order.”

References

Exit mobile version