I switched most of my log stacks from Elasticsearch to Loki a while back, and the reasoning was purely cost and operational simplicity — Loki doesn’t index full log content, only labels, which means dramatically cheaper storage at scale and far less operational overhead than running an Elasticsearch cluster. It’s not a strict upgrade (full-text search is genuinely worse), but for Kubernetes logging where you’re mostly filtering by namespace/pod/container and grepping within results, it’s an excellent fit. This article covers the full setup: Loki, Promtail (or the newer Grafana Alloy), and Grafana for visualization.
How Loki’s Architecture Differs from Elasticsearch
The key architectural decision in Loki is index-light, label-based storage. Instead of indexing every word in every log line (like Elasticsearch does), Loki only indexes metadata labels (namespace, pod, container, app) and stores the actual log content as compressed chunks in object storage (S3, GCS, or local filesystem for small setups).
Pods (stdout/stderr)
│
▼
Promtail / Grafana Alloy (DaemonSet, tails logs, attaches labels)
│
▼
Loki (indexes labels only, stores compressed log chunks)
│
▼
Grafana (LogQL queries)
This means queries filtered by label ({namespace="production", app="checkout"}) are extremely fast and cheap, while full-text search within a label set (|= "connection refused") scans the actual chunks — slower than Elasticsearch’s inverted index, but still fast enough for typical incident response at a fraction of the storage cost.
Deploying Loki
The loki-stack or standalone loki Helm chart is the standard path. For production, I use the distributed/simple-scalable deployment mode with S3-compatible object storage rather than the single-binary mode:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
# loki-values.yaml
loki:
auth_enabled: false
commonConfig:
replication_factor: 1
storage:
type: s3
bucketNames:
chunks: my-loki-chunks
ruler: my-loki-ruler
admin: my-loki-admin
s3:
region: us-east-1
schemaConfig:
configs:
- from: "2024-01-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
deploymentMode: SimpleScalable
backend:
replicas: 2
read:
replicas: 2
write:
replicas: 3
gateway:
enabled: true
kubectl create namespace logging
helm install loki grafana/loki -n logging -f loki-values.yaml
Verify:
kubectl get pods -n logging
kubectl get svc -n logging | grep loki
For quick evaluation or small clusters, single-binary mode is far simpler:
helm install loki grafana/loki -n logging \
--set deploymentMode=SingleBinary \
--set singleBinary.replicas=1 \
--set loki.storage.type=filesystem
Deploying Grafana Alloy (Log Shipper)
Grafana Alloy is the modern successor to Promtail and is what Grafana now recommends for new deployments. It runs as a DaemonSet, discovers pods via the Kubernetes API, and ships logs to Loki with labels attached automatically.
helm install alloy grafana/alloy -n logging -f alloy-values.yaml
# alloy-values.yaml
alloy:
configMap:
content: |
discovery.kubernetes "pods" {
role = "pod"
}
discovery.relabel "pod_logs" {
targets = discovery.kubernetes.pods.targets
rule {
source_labels = ["__meta_kubernetes_namespace"]
target_label = "namespace"
}
rule {
source_labels = ["__meta_kubernetes_pod_name"]
target_label = "pod"
}
rule {
source_labels = ["__meta_kubernetes_pod_container_name"]
target_label = "container"
}
}
loki.source.kubernetes "pods" {
targets = discovery.relabel.pod_logs.output
forward_to = [loki.write.default.receiver]
}
loki.write "default" {
endpoint {
url = "http://loki-gateway.logging.svc/loki/api/v1/push"
}
}
If you’re using the older but still widely deployed Promtail instead:
# promtail-values.yaml
config:
clients:
- url: http://loki-gateway.logging.svc/loki/api/v1/push
snippets:
pipelineStages:
- cri: {}
helm install promtail grafana/promtail -n logging -f promtail-values.yaml
kubectl get pods -n logging -l app.kubernetes.io/name=alloy
Connecting Grafana to Loki
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki-gateway.logging.svc
isDefault: false
If you already have Grafana running from a Prometheus setup, add this as a Helm values override to that release, or configure it manually under Connections → Data Sources → Add data source → Loki in the UI.
Writing LogQL Queries
LogQL is Loki’s query language, structurally similar to PromQL but for logs. A basic label-filtered query:
{namespace="production", app="checkout-service"}
Adding a text filter:
{namespace="production", app="checkout-service"} |= "error"
Excluding noise:
{namespace="production", app="checkout-service"} |= "error" != "healthcheck"
Parsing structured JSON logs and filtering on a field:
{namespace="production", app="checkout-service"}
| json
| level="error"
| duration > 500
Computing log volume as a rate (useful for detecting log storms or silent failures):
sum(rate({namespace="production"}[5m])) by (app)
Counting error logs over time, turned into an alertable metric:
sum(count_over_time({namespace="production"} |= "ERROR" [5m])) by (app)
Setting Up Alerts on Log Content
Loki supports Prometheus-style alerting rules directly against LogQL queries via the Ruler component:
apiVersion: v1
kind: ConfigMap
metadata:
name: loki-alert-rules
namespace: logging
data:
rules.yaml: |
groups:
- name: log-alerts
rules:
- alert: HighErrorLogRate
expr: |
sum(rate({namespace="production"} |= "ERROR" [5m])) by (app) > 10
for: 5m
labels:
severity: warning
annotations:
summary: "High error log rate in {{ $labels.app }}"
This gets mounted into the Loki ruler component (or a dedicated ruler deployment) and routes to the same Alertmanager instance your Prometheus alerts already use — one alerting pipeline for both metrics and logs.
Building Grafana Dashboards
I typically build an “Application Logs” dashboard with:
- A log panel using
{namespace="$namespace", app="$app"}with dashboard variables for namespace/app selection - A log volume time-series panel using the
sum(rate(...))query above, split by app - An error rate panel filtering on
|= "ERROR"or structuredlevel="error"depending on log format
Dashboard variables make this reusable across every service without duplicating dashboards:
{
"templating": {
"list": [
{
"name": "namespace",
"type": "query",
"query": "label_values(namespace)"
},
{
"name": "app",
"type": "query",
"query": "label_values({namespace=\"$namespace\"}, app)"
}
]
}
}
Production Considerations
- Label cardinality discipline: Never use high-cardinality values (request IDs, user IDs, timestamps) as Loki labels — this defeats the entire index-light design and can crash the ingesters. Keep those as parsed fields in log content instead, queried with
| json | field="value". - Retention: Configure
limits_config.retention_periodand object storage lifecycle rules together; Loki’s compactor handles retention enforcement. - Object storage costs: Loki is cheap relative to Elasticsearch, but S3 GET/PUT request costs still add up at high log volume — batch and compress aggressively (Loki does this by default, but tune
chunk_target_sizeif needed).
Securing Loki
Loki has no authentication of its own by default (auth_enabled: false in the config above) — it relies entirely on network-level or gateway-level access control. For any shared or multi-team cluster, this needs to be locked down:
loki:
auth_enabled: true
With auth_enabled: true, every request must include an X-Scope-OrgID header, effectively giving you basic multi-tenancy — different teams’ logs stay isolated from each other. Pair this with an authenticating proxy in front of the gateway:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: loki-gateway-ingress
namespace: logging
annotations:
nginx.ingress.kubernetes.io/auth-url: "https://auth.internal.example.com/verify"
spec:
tls:
- hosts:
- loki.internal.example.com
secretName: loki-tls
rules:
- host: loki.internal.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: loki-gateway
port:
number: 80
Grafana’s Loki datasource then needs the appropriate X-Scope-OrgID header configured per-team if you’re running true multi-tenancy, or a single shared tenant ID if you’re just using auth_enabled as a basic access gate rather than full isolation.
Migrating from Elasticsearch/Kibana to Loki
If you’re moving off an existing ELK-based logging setup (see the companion Kibana article), a few practical notes from having done this migration a few times:
- Run both stacks in parallel during a transition window — Promtail/Alloy can ship the same log stream to both Loki and Elasticsearch simultaneously via multiple Filebeat outputs or a fan-out in the OpenTelemetry Collector, letting teams validate Loki dashboards against known-good Kibana results before fully cutting over.
- Rebuild dashboards rather than trying to directly port Kibana saved searches — LogQL and Elasticsearch’s Query DSL are different enough that a straight translation rarely produces equivalent results, especially for anything using Elasticsearch aggregations.
- Budget time for the label-cardinality cleanup — teams coming from Elasticsearch (where indexing everything is normal) often carry that habit into early Loki label design, which causes exactly the ingester problems described above.
Troubleshooting Loki Ingestion Issues
Logs not appearing despite Alloy/Promtail running fine. Check the write path directly:
kubectl logs -n logging -l app.kubernetes.io/name=alloy --tail=50 | grep -i error
kubectl port-forward -n logging svc/loki-gateway 3100:80
curl -s "http://localhost:3100/loki/api/v1/query?query={namespace=\"production\"}" | jq .
“per-stream rate limit exceeded” errors. This means too much log volume is being pushed under labels that create too few distinct streams — often a symptom of insufficiently granular labels (everything under one app label) rather than the cardinality-too-high problem; the two errors look similar but have opposite root causes and opposite fixes.
Ingesters running out of memory. Almost always cardinality-related — check the number of active streams:
curl -s "http://localhost:3100/metrics" | grep loki_ingester_memory_streams
A rapidly growing stream count over time (rather than plateauing) is the clearest sign of a high-cardinality label leaking into your label set.
Common Mistakes
- Adding high-cardinality labels like
pod_iportrace_idas Loki labels instead of parsed log fields — this is the single most common Loki misconfiguration and causes ingester memory pressure. - Running single-replica Loki in production with local filesystem storage, losing all logs on pod eviction.
- Forgetting
for: 5m(or similar) on log-based alert rules, causing noisy alerts from transient blips.
Summary
Loki trades full-text indexing power for dramatically lower cost and operational simplicity by indexing only labels and storing log content as compressed chunks. On Kubernetes, the practical setup is Loki (simple-scalable mode with S3 backing) plus Grafana Alloy or Promtail as the DaemonSet shipper, queried through LogQL in Grafana — often sitting right next to your existing Prometheus dashboards in the same tool. The one discipline that matters most is keeping label cardinality low; get that wrong and you’ll fight ingester stability the whole way.
