A lot of Kubernetes monitoring guides default to the Prometheus/Grafana stack, but plenty of organizations already run Splunk as their enterprise-wide observability platform and want their Kubernetes clusters feeding into it rather than standing up a parallel toolchain. In this guide, I’ll walk through setting up full Kubernetes monitoring with Splunk — logs, metrics, and traces — using the Splunk OpenTelemetry Collector, which has become the standard, supported path for this integration.
Why Splunk for Kubernetes
Splunk excels at correlating data across an entire organization — infrastructure, applications, security events — in one searchable index. If your security team already lives in Splunk for SIEM purposes, routing Kubernetes observability data there too means one pane of glass, one query language (SPL), and one retention/compliance policy, instead of maintaining Splunk alongside a separate Prometheus stack.
Kubernetes Architecture Context: What You’re Actually Monitoring
Before wiring up any tool, it helps to be clear on what data sources exist in a cluster:
- Container logs: stdout/stderr from every container, written to disk on each node by the container runtime.
- kubelet /metrics/cadvisor: Per-container CPU, memory, disk, and network usage.
- kube-state-metrics: Cluster-level object state (Deployment replica counts, Pod phase, node conditions) — this is not the same as resource usage; it’s object metadata as metrics.
- API server metrics: Request latency, error rates — critical for cluster health.
- Application metrics: Whatever your app exposes, often in Prometheus exposition format.
A complete monitoring setup pulls from all of these.
Step 1: Install the Splunk OpenTelemetry Collector via Helm
Splunk’s officially supported path for Kubernetes is the Splunk Distribution of the OpenTelemetry Collector, deployed as a Helm chart that runs both a node-level agent (DaemonSet) and a cluster-level receiver (Deployment).
helm repo add splunk-otel-collector-chart https://signalfx.github.io/splunk-otel-collector-chart
helm repo update
Create your values file:
# splunk-otel-values.yaml
splunkObservability:
realm: us1
accessToken: "<YOUR_SPLUNK_ACCESS_TOKEN>"
clusterName: production-cluster
logsEngine: otel
splunkPlatform:
endpoint: "https://splunk-hec.yourcompany.com:8088/services/collector"
token: "<YOUR_HEC_TOKEN>"
index: "kubernetes"
metricsIndex: "k8s_metrics"
insecureSkipVerify: false
metricsEnabled: true
tracesEnabled: true
logsEnabled: true
Install it:
helm install splunk-otel-collector splunk-otel-collector-chart/splunk-otel-collector \
-n monitoring --create-namespace \
-f splunk-otel-values.yaml
Verify:
kubectl get pods -n monitoring
Expect to see an agent DaemonSet Pod on every node, plus a k8s-cluster-receiver Deployment Pod.
Step 2: Understand the Two Collector Roles
- Agent (DaemonSet): Runs on every node, tails container log files directly off disk, scrapes kubelet/cAdvisor metrics for that node’s Pods, and receives traces from local application SDKs.
- Cluster Receiver (single Deployment): Watches the Kubernetes API for cluster-wide object state (this is where kube-state-metrics-equivalent data comes from) so you’re not duplicating that collection on every node.
This split matters for resource planning — the agent’s resource requests scale with node count, while the cluster receiver stays a single low-overhead Pod regardless of cluster size.
Step 3: Verify Data Is Reaching Splunk
In Splunk Search:
index=kubernetes | stats count by k8s.namespace.name, k8s.pod.name
For metrics (if using Splunk Infrastructure Monitoring / Observability Cloud):
index=k8s_metrics metric_name="k8s.pod.cpu.utilization"
| stats avg(value) by k8s.pod.name
If nothing shows up after a few minutes, check the collector’s own logs first:
kubectl logs -n monitoring daemonset/splunk-otel-collector-agent
Look for HEC connection errors — the most common early failure is a wrong token or an unreachable HEC endpoint due to network policy or firewall rules.
Step 4: Configure HTTP Event Collector (HEC) on the Splunk Side
If you’re managing Splunk yourself (not Splunk Cloud), make sure HEC is enabled and a token exists with access to your target index:
# On the Splunk instance
splunk enable listen 8088 -auth admin:changeme
In Splunk Web: Settings → Data Inputs → HTTP Event Collector → New Token, scoped to the kubernetes and k8s_metrics indexes you created.
Step 5: Add Kubernetes Metadata Enrichment
By default, the collector enriches every log line and metric with Kubernetes metadata (namespace, pod name, labels, node) via the k8sattributes processor, which is already configured in the chart. You can extend it to pull specific custom labels your teams use for ownership tagging:
agent:
config:
processors:
k8sattributes:
extract:
labels:
- tag_name: team
key: team
from: pod
- tag_name: environment
key: env
from: pod
This means every log and metric arriving in Splunk is automatically tagged with which team owns the Pod that generated it — critical for building dashboards and alerts scoped by team without manual tagging.
Step 6: Building Dashboards and Alerts
Once data is flowing, build a basic health dashboard in Splunk with panels like:
# Pod restart count in last hour
index=k8s_metrics metric_name="k8s.pod.restart_count"
| timechart span=5m max(value) by k8s.pod.name
# Error rate from application logs
index=kubernetes k8s.namespace.name="production" log_level="ERROR"
| timechart span=1m count
Set up alerts on conditions like sustained high restart counts, node NotReady status, or PVC usage crossing a threshold — all queryable the same way you’d alert on any other Splunk data source.
Security Considerations
- Use a dedicated Kubernetes ServiceAccount with RBAC scoped only to read (
get,list,watch) on the resources the collector needs — never grant it write access. - Store the Splunk access token and HEC token as Kubernetes Secrets, referenced via
envFrom, never hardcoded in values files committed to Git. - Enable TLS between the collector and your Splunk HEC endpoint (
insecureSkipVerify: falsein production, always).
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: splunk-otel-collector
rules:
- apiGroups: [""]
resources: ["pods", "nodes", "namespaces", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["get", "list", "watch"]
Performance and Cost Optimization
- Filter noisy logs at the collector, not after ingestion — dropping health-check log spam before it hits Splunk saves significant licensing cost, since Splunk pricing is typically volume-based.
- Downsample high-cardinality metrics where per-second granularity isn’t needed.
- Use index-time field extraction sparingly — prefer search-time extraction where possible to reduce indexing overhead.
Common Mistakes
- Forgetting to set
clusterNameuniquely per cluster — without it, data from multiple clusters gets conflated in Splunk, making dashboards useless in multi-cluster environments. - Not setting resource limits on the agent DaemonSet, letting it consume unbounded memory on log-heavy nodes.
- Sending 100% of logs from noisy sidecars (like service mesh proxies) without filtering, drowning out application signal.
High Availability
Run the cluster receiver Deployment with at least 2 replicas behind leader election (the chart supports this natively) so a single Pod restart doesn’t create a gap in cluster-level metric collection. The DaemonSet agent is inherently HA per-node since each node has its own instance.
Summary
Splunk can absolutely serve as your primary Kubernetes observability backend, and the Splunk OpenTelemetry Collector Helm chart is the well-supported way to get there — combining a DaemonSet agent for node-local logs/metrics/traces with a cluster receiver for object-level state. Get HEC configured correctly first, verify data with simple SPL queries, then layer in metadata enrichment, dashboards, and alerts once you trust the pipeline.
